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ABSTRACT 



This thesis reviewed two different approaches to benefit analysis, 
benefit comparison and user satisfaction, that could be applied to the 
evaluation of proposed information systems which are under 
consideration for acquisition by the federal government. Currently the 
General Services Administration only recommends that present value 
analysis methods be used in the analysis of alternatives even though 
the GSA specifies that non-cost factors should be evaluated in such an 
analysis. 

Different benefit comparison and user satisfaction methods are 
reviewed for their particular advantages and disadvantages. A 
discussion is given on how selected methods of each approach may be 
used within the federal government for the evaluation of alternatives. 
Suggestions are made for ways of conducting a more complete analysis 
of alternatives through incorporating present value analysis, benefit 
comparison of non-cost factors, and analysis of user satisfaction, into 
one comprehensive analysis. 
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I. 



INTRODUCTION 



A. BACKGROUND 

The need for benefit analysis methods capable of being 
applied to the evaluation of proposed information systems 
has long been recognized. Twenty years ago Chervany and 
Dickson (Chervany and Dickson, 1970 ) asserted then that it 
was time to re-evaluate the decision-making process 
involving the development of information system 
alternatives. In the mean time the demand for information 
systems, and hence the demand for benefit analysis of these 
systems, has grown substantially. Abdel -Hamid and Madnick 
(Abdel -Hamid and Madnick, 1989) estimate the increase in 
demand at a hundredfold over the last two decades. Kitfield 
(Kitfield, 1989) estimated the demand for avionic software 
was increasing 25 percent per year. This trend may be even 
more true for the Department of Defense (DoD) as one expert 
has described what he calls the "military software crisis" 
(Kitfield, 1989) . Kitfield describes this crisis as, 

"runaway demand and a profound shortage of software 
programmers" (Kitfield, 1989) . The problem of selecting an 
information system alternative may now be focused more on 
which alternative to invest in first. This situation 
reinforces the need for benefit analysis to prioritize 
software projects. 
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B. OBJECTIVES 



The objective of this thesis, is to determine what 
methodologies can be used to analyze the benefits to be 
gained from a proposed information system. More 
specifically the objective is to examine what current 
benefit analysis methods can be used in the evaluation of 
different competitive bids submitted in response to a 
solicitation from the federal government for a proposed 
information system. Of special interest is how intangible 
benefits of advanced information technology are treated in 
such an analysis. 

C. RESEARCH QUESTIONS 

The research concerns of this thesis are really two 
fold. First, which benefit analysis methods can be employed 
to compare the benefits of two competing proposed 
information systems to determine the alternative that is the 
most advantageous to the government. Secondly, which 
benefit analysis methods are capable of calculating a 
monetary value of the benefits from a complex information 
system prior to that system's development. 

D. SCOPE AND LIMITATIONS 

This study will focus on benefit analysis methods that 
can be used in the evaluation of alternatives which is 
conducted in the process of a federal acquisition of an 
information system. Benefit analysis methods will also be 
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examined that can be incorporated into a larger cost-benefit 
analysis. The subject of risk assessment, in relation to 
information systems, is included under benefit analysis of 
proposed information systems for the purpose of this thesis. 
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I I . METHODOLOGY 



A. SELECTING STUDIES OF BENEFIT ANALYSIS 

1. Choosing a Medium from which to Select Studies 
Published articles were chosen as the source of 

studies on benefit analysis in both existing and proposed 
information systems. Studies on existing systems were 
included to gain a better insight into the subject of 
benefit analysis of proposed information systems. Only 
articles of benefit analysis in English-language journals, 
in the field of management information systems and the 
related field of management science, were surveyed. 

2. Choosing a Time Span of the Studies to Survey 
The work on this literature review began in June, 

1990 and was completed in January, 1991. The last articles 
that were excepted for inclusion were published in the 
calendar year of 1990. The year 1970 was selected as the 
starting point for the bibliographical search of this 
literature review. However, a time limit was not placed on 
articles or supporting publications that could be 
incorporated into the research based on references in 
articles found in the main bibliographical search or books 
that supported the underlying concepts of an article. 
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3. Search for Key Words 

a. The Initial Bibliographical Search 

An initial search was conducted manually of 
Harvard Business Review and Management Information Systems 
Ouarterl v to find key words that could be used in the search 
of on-line bibliographic data. These journals were used 
because of their history of articles published in the field 
of management information systems. The table of contents of 
every issue between 1970 and 1990 was manually reviewed. If 
a title appeared relevant to the subject of benefit analysis 
of information systems then the article was reviewed. After 
reading the article and determining that indeed it did 
pertain to the subject, the article was then copied and 
saved for later referral. 

b. Key Words Found 

The search for key words identified 19 articles 
from these two journals. From the initial search it could 
also be concluded that the authors of the 19 articles chose 
diverse key words to describe their articles. Key words 
included : 

- Cost/Benefit Analysis 

- Cost/Benefit Calculations 

- Information Economics 

- Information System Investment 

- Information Technology Investment 

- Value Analysis 
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As the dissimilarity of key words found in the initial 
search was discovered, it was determined that a single on- 
line search with a single set of key words would not 
sufficiently identify the number of articles desired on the 
subject of benefit analysis of information systems. The 
problem was that the key words that were found were just to 
diverse for the on-line system to be effective. 

4. Manual Bibliographical Search 

As a result of the difficulties with key words, a 
manual bibliographical search was undertaken as opposed to 
an on-line search. This main search was conducted in the 
same manner as the preliminary search. A list of eight 
additional journals was gathered based on their reputation 
for publishing studies in the field of management 
information systems. The journals searched manually were: 

- Communications of the ACM * 

- Data Base 

- Decision Sciences 

- Harvard Business Review 

- Information and Management 

- Journal of Management Information Systems 

- Management Science 

- Management Information Systems Quarterly 

- Omega 

- Sloan Management Review 

* ACM: Association for Computing Machinery 
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a. Conduct of the Manual Search 



Once the list of journals was established the 
search was conducted systematically. Journal by journal, 
the table of contents for each issue available, during the 
time period of 1970 to 1990, was examined for a possible 
article relevant to the subject of interest. Articles with 
fitting titles were copied for latter reference if after 
being reviewed it was concluded that they were appropriate, 
h. Other Sources of Literature 

Other sources of reference were considered to 
clarify certain points brought up in the articles that had 
been initially collected and reviewed. The references of 
the articles located in the main search were used to 
identify further articles that could pertain to the subject 
of this study. These sources were located through on-line 
bibliographical searches by either author, title, or 
subject. The on-line search was useful to locate these 
additional sources within the library for in this instance 
very specific: author names, and titles, were used to query 
the on-line system for exact locations of individual works. 

A final source of literature that comprises the foundation 
of this thesis came from the courses that this author was 
exposed to at the Naval Postgraduate School . A 
comprehensive inventory of the literature found and used for 
this thesis is presented in the bibliography. 
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B. SORTING THE WORKS ON BENEFIT ANALYSIS 



In combination, the preceding search methods yielded a 
total of 50 references that were considered for possible 
contribution to the study of benefit analysis of proposed 
information systems. These references were organized into 
three groups: benefit comparison, user satisfaction, and 
miscellaneous literature. 

1. Benefit Comparison 

The articles in this group are distinguished from 
the others by the condition that they present methods of 
analyzing benefits by assessing various system attributes. 
The methods discussed under benefit comparison are: the 
additive weight model, the eigenvector model; and the multi- 
attribute utility model. Chapter IV is dedicated to benefit 
comparison . 

2. User Satisfaction 

The references in this group, in contrast to the 
groups previously mentioned, propose to analyze benefits not 
through an objective indicator, but through a survey to 
determine the subjective satisfaction that users have with 
their respective information system. The procedures of this 
method incorporate various formulas and models to interpret 
the responses of user satisfaction questionnaires. User 
satisfaction is the subject of Chapter V. 



8 



3. Miscellaneous Literature 



The last group that references were organized into 
was miscellaneous. This group contained articles on the 
importance of information systems and their predominance 
throughout the business world and DoD. Articles were also 
placed in this group that stressed the value of information 
and the need for benefit analysis of information systems. 
Articles on the competitive advantage to be gained from 
information systems were also placed in the miscellaneous 
group. In spite of the expressed need for benefit analysis, 
the group of miscellaneous literature did not contribute to 
any specific benefit analysis technique. As such, the 
references in this group were largely used as background 
material that stressed the importance of information system 
benefit analysis and that put benefit analysis in 
perspective in relation to analysis of alternatives and the 
rest of the development life cycle for information systems. 
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Ill . 



BACKGROUND 



A. GUIDELINES FOR FEDERAL ADP ACQUISITIONS 

The Brooks Act, public law 89-306, established the U.S 
General Services Administration (GSA) as the procurement 
authority for ADP equipment acquisitions within the federal 
government. The Act instructs the GSA to "provide for the 
economic and efficient purchase, lease, and maintenance of 
automated data processing equipment by federal agencies" 
(U.S. GSA, 1990). This authority was increased through the 
enactment of public law 99-500 to also include the federal 
acquisition of such things as; software, firmware, and 
computer related contracted services to include programming 
and support services. In fulfillment of their 
responsibility the GSA publishes both procedural guides, 
such as A Guide for Requirements Analysis and Analysis of 
Al ternatives . and the federal information resources 
management regulation (FIRMR). (U.S. GSA, 1990) 

The Warner Amendment, as enacted under public law 99- 
500, did release DoD from the jurisdiction of the Brooks Act 
for certain ADP functions that dealt with national security 
or military operations. However, the Warner Amendment did 
not release DoD from the conditions of the Brooks Act for 
ADP acquisitions that deal with a vast number of other 
functions such as logistics and administrative activities. 
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As such the DoD must abide by FIRMR when making a great 
number of ADP and information system acquisitions. 

B. GSA POLICY FOR THE ANALYSIS OF ALTERNATIVES 

In the process of making a federal acquisition of ADP 
equipment or systems the GSA requires that an analysis of 
alternatives be performed. The objective of this analysis 
is to determine the most advantageous alternative to the 
government. As a part of this analysis of alternatives the 
GSA has mandated that both cost and non-cost factors are to 
be evaluated. The specific cost and non-cost factors that 
should be considered in this type of analysis have been 
determined by the GSA and are listed, and described, in A 
Guide for Requirements Analysis and Analysis of Alternatives 
(U.S. GSA, 1990). The cost factors are: 
conversion 
personnel 
supplies 
energy 
maintenance 
space 

administrative costs of contracting 
contract price 

The cost factors above are all quite tangible and relatively 
easy to evaluate in the course of an analysis of 
alternatives. In fact the GSA has recommended three 
different methods for use in the analysis of cost factors: 



11 



present value analysis, net present value, and benefit-cost 
ratio. (U.S. GSA,1990) 

C. ANALYSIS OP COST FACTORS 
1. Description 

Of the techniques recommended by GSA for the 
analysis of cost factors, present value analysis is common 
to capital investment and calculates the present worth of a 
monetary amount to be received in the future. Present value 
analysis takes into account the time value of money, or the 
concept that a dollar is worth more today than a year from 
today. There are several different approaches to present 
value analysis to included: present value of a single future 
payment, present value net benefits, and present value for 
competing alternatives. 

a. Present Value of a Single Future Payment 

In Garrison (1988), present value analysis is 
described in the context of a the receipt of a single 
amount, or benefit, in the future. The formula for 
calculating present value of a one time payment in the 
future is given in Formula 1. (Garrison, 1988) 



(1+r)" (1) 

In Formula 1, P represents the present value of 
the benefit in monetary terms. P is the monetary value of 
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the benefit to be received in the future. r is the interest 



rate and n is the number of years in the future that the 
benefit will be received. (Garrison, 1988) 
b. Present Value Net Benefits 

Thompson (1980) examines present value analysis 
in terms of discovering the sum of all the systems results, 
both good and bad, or in other words both liabilities and 
assets. In his work Thompson uses the expression "present- 
valued compensating variation" when referring to the 
difference between the cost and benefit of a particular 
attribute. Thompson's formula for present-valued net 
benefits is given in Formula 2. (Thompson, 1980 ) 



n 



2=1J=1 



cv, 



m 

(l+d)^ 



U. 



( 2 ) 



In Formula 2, CV^j is the compensating variation 
for benefit j on the user i. d is the discount rate. y is 
the number of years until the user i is affected by benefit 
j. m is the number of different benefits. n is the number 
of users benefitted by the system. (Thompson, 1980) 

Thompson's present-valued net benefits method 
should not be confused with the net present value (NPV) 
method. As recommended by GSA, the NPV is equal to 
difference between the total present value of all of the 
benefits and the total present value of all of the costs. 
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NPV, unlike present-valued net benefits, does not attempt to 
sum the differences between costs and benefits of each 
characteristic of an information system. (U.S. GSA,1990) 
c. Present Value for Competing Alternatives 
In NAVDAC PUB 15 (Dec, 1980) present value 
analysis is used to evaluate alternatives based on lowest 
cost. In this application of present value analysis, the 
benefits of all alternatives being considered must be equal 
and a selection is made by determining the lowest present 
value of the costs of the different alternatives. Also, by 
this method, the service lives of the alternatives must be 
finite and equal among alternatives, or be placed in equal 
terms. In computing present values the NAVDAC method uses 
standard tables of discount factors in which the present 
value of a dollar can be found for both a single cash flow 
and cumulative uniform cash flows. Uniform here being the 
same amount paid every year. This form of the present value 
may be easier to conceptualized if placed in the context of 
hardware acquisitions. (NAVDAC PUB 15,1980) 

2. Example of Present Value Analysis 

In this example two high capacity printers are under 
consideration for purchase. Both of the printers have the 
same exact capabilities, such as the same output speeds, 
font styles, and buffer sizes. One of the printers, 
alternative A, has a service life of six years. The other, 
alternative B, has a service life of three years. In this 
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situation the NAVDAC method of present value analysis can be 
applied for the benefits are equal and the service lives can 
be placed in equal terms. (NAVDAC PUB 15,1980) 

To further define the example, alternative A costs 
$10,000 to purchase and has annual reoccurring cost of 
$4,000. Alternative B costs $8,000 to purchase and has 
annual reoccurring costs of $5,000. Both alternatives are 
not expected to retain any value at the end of the economic 
lives. (NAVDAC PUB 15,1980) 

a. Present Value Using a Six Year Service Life 

The present value for the two alternatives using 
a six year service life would be found as follows. For 
alternative A the purchase price would be added to the 
reoccurring expenses multiplied by the factor for a six year 
cumulative uniform series. In alternative B two printers 
have to be purchased so that their combined service life 
equals six years. The present value of alternative B is the 
purchase price for the first printer added to the 
reoccurring expenses multiplied by the factor for a six year 
cumulative uniform series and then added to the purchase 
price of the second printer multiplied by the factor for a 
single payment in three years. Algebraically the present 
values determination of the alternatives is shown below. 
(NAVDAC PUB 15,1980) 

A = $10,000 + ($4,000 X 4.570) = $28,280 

B = $8,000 + ($5,000 X 4.570) + ($8,000 x . 788 ) =$37 , 154 
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b. Present Value Using a Three Tear Service Life 
Had a three year service life been used in the 
comparison only one purchase in alternative B would have to 
have been considered. However, as alternative A would have 
half of its service life remaining at the end of three 
years, then half of its purchase price, multiplied by the 
factor for a single cash flow in three years, would have to 
be deducted from the calculation to represent the remaining 
value. Also, the reoccurring expenses multiplied by the 
factor for a three year cumulative uniform series, not a six 
year series. Algebraically the present values determination 
of the alternatives using this three year service life is 
given below. (NAVDAC PUB 15,1980) 

A = $10,000 + ($4,000 X 2.609) - ($5,000 x . 788 ) =$16 , 496 
B = $8,000 + ($5,000 X 2.609) = $21,045 
3. Advantages & Disadvantages of Present Value Analysis 
A key advantage of the present value technique is 
that organizations are familiar with it for the technique is 
used to make other capital budgeting decisions 
(Garrison, 1988) . Another advantage is that present value 
analysis is a simple means of comparing alternatives when 
the benefits and project lives are the same (NAVDAC PUB 
15,1980). Then possibly the most compelling advantage of 
present value analysis is that it is one of the three cost 
analysis methods recommended by the GSA for use in the 
analysis of alternatives. This analysis being required for 
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each federal acquisition of ADP equipment or information 
systems . 

Present value analysis alone is seen by some as ill 
suited for the analysis of some types of information 
systems, such as decision support systems (Keen, 1981). The 
technique is not recommended in this situation for it is 
difficult to quantify the benefits of such a system 
capabilities such as AD-HOC analysis (Keen, 1981). Present 
value analysis, as described above, is also criticized for 
it does not take into account risk or the probability of the 
benefit occurring (Couger et al,1982). Possibly for these, 
or similar reasons GSA has also required that non-cost 
factors be included in the analysis of alternatives. 

D. ANALYSIS OF NON-COST FACTORS 

The non-cost factors that the GSA has specified that are 
to be included in an analysis of alternatives have been 
separated by GSA into two groups; functional factors and 
risk factors. The functional factors include: (U.S. 

GSA, 1990) 

obsolescence 

availability 

reliability 

maintainability 

expandability 

flexibility 

security 
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privacy 

personnel impacts 
user acceptance 
accountability 

The risk factors are: (U.S. GSA,1990) 

financial 

technical 

schedule 

The inclusion of these functional and risk factors into 
an analysis of alternatives offsets the disadvantages of 
conducting an analysis solely on the basis of cost factors. 
Those disadvantages namely being; the lack of addressing 
risk, and the inappropriate nature of attempting to apply 
cost-benefit analysis methods to benefits that are difficult 
to quantify in dollar terms. As such, there should be a 
clear advantage of conducting an analysis of alternatives 
that incorporates both cost and non-cost factors. However, 
even though GSA does directs that the functional and risk 
factors be included in the analysis of alternatives, the GSA 
guides do not recommend or mension any non-cost analysis 
method which should be used in such an analysis. (U.S. 

GSA, 1990) 
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IV. ANALYSIS METHODS FOR NON-COST FACTORS 



There are existing benefit comparison methods that may 
well suited for fulfilling the GSA requirement for non-cost 
analysis. That analysis, of course, being part of an 
analysis of alternatives leading up to the federal 
acquisition of ADP equipment or information systems. These 
benefit comparison methods not only are useful in 
contrasting the benefits of the various alternatives, they 
can also be used for making a selection from proposed 
alternatives based on non-cost factors. The non-cost 
factors in this context may be described as attributes of 
information systems and may be analyzed as well as measured 
as such by the different benefit analysis methods. The 
supporters of objective quantification methods believe that 
analysts, who describe the benefits of information systems 
as completely intangible, are not putting forth the effort 
to find measures for the benefits (Couger et al,1982). The 
benefit comparison methods that will be addressed are: the 

additive weight model, the eigenvector model, and the multi- 
attribute utility model. 

A. THE ADDITIVE HEIGHT MODEL 
1. Description 

The additive weight model consists of three steps; 
determining weights, scoring alternatives, and selecting the 
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best alternative. Prior to using the model however, the 
benefits must be defined and broken down into the separate 
attributes or characteristics of the information system. 

Such attributes could include: memory capacity, calculation 
speed, or the manufacturer's reliability. These attributes 
can be divided by categories, such as hardware and software, 
or listed together in one group. The attributes are not 
required to be independent of each other (Shoval and 
Lugasi , 1987 ) . Defining the attributes establishes common 
references in which the additive weight model can be 
implemented by analysts and decision makers alike. (Shoval 
and Lugasi, 1987) 

a. The Three Steps of the Additive Height Model 

The first step determines attribute weights for 
each attribute. These may be obtained through point 
allocation where a decision maker assigns a number between 0 
and 1 to each attribute to reflect its relative importance 
(Schoemaker and Waid,1982). The sum of all the attribute 
importance weights must equal 1. If attributes are divided 
into categories then category importance weights must also 
be selected and their sum must also equal 1. 

The second step, scores alternatives, by having 
the decision maker evaluate each alternative and assess a 
rating for each attribute. This is done one attribute at a 
time and only the attributes receive a rating, not the 
categories or the overall system. The final step, selection 
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of the best alternative, is made by choosing the information 
system that maximises the summation of attribute scores 
multiplied by the respective attribute and category weights. 
This step is expressed in Formula 3 and demonstrated in 
Table 3. (Shoval and Lugasi,1987) 

b. The Additive Height Formula 

Formula 3 is the additive weight formula. When 
using this formula four conditions must hold true. First, 
the sum of the category weights must equal 1. Second, 
attribute weights range from 0 to 1. Third, attribute 
scores must be less than or equal to 1. Finally, the sum of 
the attribute importance weights must equal 1. (Shoval and 
Lugasi ,1987 ) 

The following notation is used in Formula 3 and 
Tables 1 through 3. Zj expresses the category importance 
weights of m categories. represents attribute importance 
weights of n attributes. Finally V^j symbolizes the ith 
attribute score for alternative j. (Shoval and Lugasi, 1987) 

JD 

(3) 



2. Example of the Additive Weight Model 

In Shoval and Lugasi (1987) an example of the 
additive weight model is given where there are four 
alternatives to select from and 14 attributes to be 
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evaluated which were split into three categories. The four 
proposed systems are alternatives A through D. The three 
categories are hardware, software, and support. The 
attributes, listed by category, are given in Table 1 along 
with their respective importance weights. Alternative 
scores by attribute are given in Table 2. (Shoval and 
Lugasi ,1987 ) 



Table 1. Category and Attribute Importance Weights 



Category Attribute 


Weight 


Z1 


Hardware 


.30 




XI Memory Capacity 


.25 




X2 Calculation Speed 


.25 




X3 I/O Speed 


.07 




X4 Equipment Reliability 


.25 




X5 Flexibility for Expansion 


.18 


Z2 


Software 


.30 




X6 Availability for Scientific 


Software .30 




X7 Flexibility to Changes 


.30 




X8 Software Performance 


.40 


Z3 


Support 


.40 




X9 Manufacturer's Reliability 


.20 




XIO Supplier's Reliability 


.20 




Xll Supply Time 


.10 




XI2 Quality of Hardware Support 


.20 




XI3 Quality of Software Support 


.20 




X14 Quality of Documentation 


.10 
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Table 2. Alternative Scores by Attribute 



Attributes 


A 


Alternatives 
B C 


D 


XI 


0.8 


0.8 


0.8 


1.0 


X2 


0.8 


0.8 


0.8 


0.6 


X3 


0.8 


0.8 


0.8 


1.0 


X4 


1.0 


1.0 


0.7 


0.9 


X5 


1.0 


0.9 


0.9 


0.8 


X6 


0.8 


0.8 


0.8 


0.6 


X7 


1.0 


1.0 


1.0 


1.0 


X8 


0.9 


0.9 


0.8 


0.9 


X9 


1.0 


1.0 


0.8 


0.9 


XIO 


0.9 


0.9 


0.7 


0.8 


XI 1 


0.9 


0.7 


1.0 


0.9 


XI 2 


1.0 


1.0 


0.8 


1.0 


XI 3 


0.9 


0.9 


1.0 


0.8 


XI 4 


1.0 


1.0 


1.0 


0.9 



Table 3. Calculation of the Weighted Score for 
Alternative A 



Attribute 


Category x 
Weight 


Attribute 

Weight 


X Attribute 
Score 


= Weighted 
Score 


XI 


0.3 


.25 


0.8 


.0600 


X2 


0.3 


.25 


0.8 


.0600 


X3 


0.3 


.07 


0.8 


.0168 


X4 


0.3 


.25 


1.0 


.0750 


X5 


0.3 


.18 


1.0 


.0540 

.2658 


X6 


0.3 


0.3 


0.8 


.0720 


X7 


0.3 


0.3 


1.0 


.0900 


X8 


0.3 


0.4 


0.9 


.1080 

.2700 


X9 


0.4 


0.2 


1.0 


.0800 


XIO 


0.4 


0.2 


0.9 


.0720 


XI 1 


0.4 


0.1 


0.9 


.0360 


XI 2 


0.4 


0.2 


1.0 


.0800 


XI 3 


0.4 


0.2 


0.9 


.0720 


X14 0.4 

Total Weighted Score 


0.1 


1.0 


.0400 

.3800 

.9158 
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From Formula 3 and Tables 1 through 3, alternative 

A, with a score of 0.9158, has the highest total weighted 
score of the four alternatives. Alternative A would thus be 
the information system selected by the additive weight 
model. The weighted scores of alternatives B, C, and D are 
0.9024, 0.8399, and 0.8557 respectively. (Shoval and 
Lugasi , 1987 ) 

3. Advantages and Disadvantages of Additive Height 

Besides being simple in nature the additive weight 
model also has the advantage that sensitivity analysis, or 
receptiveness to change, can be measured for the weights 
assigned to the various attributes and categories. The 
disadvantage is that this model cannot check the consistency 
of evaluators; nor does it consider risk or uncertainty. 
Assigning weights and scoring alternatives is also a 
subjective appraisal of information system benefits. 

(Shoval and Lugasi, 1987) 

B. THE EIGENVECTOR MODEL 
1. Description 

Building on the additive weight model the 
eigenvector model provides a means for determining the 
attribute and category weights as well as scoring the 
alternatives. Final system selection in the eigenvector 
model is made using Formula 3 of the additive weight model. 
In the eigenvector model attribute weights are found by a 
process which starts with pairwise comparisons. A decision 
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maker completes a matrix in which both the number of columns 
and rows equal the number of attributes being evaluated. 

For instance, if four attributes were being examined, then a 
4x4 matrix would be required. Each cell of the matrix 
represents a comparison between two of the attributes. A 
nine-point scale is used in the model with 1 representing 
equality between attributes and 9 depicting absolute 
preference of one attribute over the other. In the matrix 
in Figure 1, attribute xl is favored 3/2 times to attribute 
x5. The consistency of the decision maker can be verified 
through an inspection of inverse relationships. Here x5 is 
favored 2/3 times to xl so the decision maker is being 
consistent. (Shoval and Lugasi,1987) 

Figure 1. Sample Matrix 





Xl 


X2 


X3 


X4 


X5 


Xl 


1 


1 


5/2 


1 


3/2 


X2 


1 


1 


5/2 


1 


3/2 


X3 


2/5 


2/5 


1 


1/3 


1/2 


X4 


1 


1 


3 


1 


3/2 


X5 


2/3 


2/3 


2 


2/3 


1 



The eigenvector of this matrix is then computed for 
the maximum eigenvalue. This eigenvector is then normalized 
so that the sum of the elements in the vector equals 1. 
Computing and normalizing these eigenvectors can be done 
through the use of a computer package such as EISPACK, which 
is a package of fortran IV programs (Goos and 
Hartmanis , 1977 ) . The elements in this final form of the 



25 



eigenvector are the respective attribute weights. Category 
weights are then determined in a similar fashion with the 
dimensions of the matrix equal to the number of categories. 
(Shoval and Lugasi,1987) 

To score the alternatives by attribute, a matrix of 
pairwise comparisons is used. This time the nuhd^er of 
matrices used is equivalent to the number of attributes. 

The dimensions of each of these matrices is set to the 
number of alternatives. The elements of the normalized 
eigenvector are the solution. Here they represent the 
scores of the alternatives for the respective attributes. 
(Shoval and Lugasi,1987) 

2. Example of the Eigenvector Model 

The situation used in this example is the same as 
that used above for the additive weight model: four 
alternatives, three categories, and 14 attributes. The four 
alternatives are still labeled A through D and the 
categories and attributes remain the same from Table 1 
above. In Figure 2 of Appendix A the matrices for 
determining attribute weights are displayed. Figure 3 of 
Appendix A is the matrix of pairwise comparisons of the 
three categories. In Figure 4 of Appendix A the 14 matrices 
are presented that were used to define the alternative 
scores by attribute. Finally, Table 4 contains the category 
and attribute weights, and lists the weighted scores for the 
each alternatives which is used to produce their respective 
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ranking as determined by the eigenvector model. (Shoval and 
Lugasi ,1987 ) 



Table 4. Scores and Ranking of Alternatives 



Attribute 


Weight 


A 


Alternatives 
B C 


D 


Z1 


.333 










XI 


.244 


.222 


.222 


.222 


.333 


X2 


.244 


.286 


.286 


.286 


.143 


X3 


.091 


.211 


.211 


.199 


.378 


X4 


.253 


.316 


.316 


.158 


.210 


X5 


.169 


.286 


.286 


.286 


.143 


Z2 


.333 










X6 


.400 


.286 


.286 


.286 


.143 


X7 


.200 


.250 


.250 


.250 


.250 


X8 


.400 


.273 


.273 


.273 


.182 


Z3 


.333 










X9 


.352 


.375 


.250 


.125 


.250 


XI 0 


.235 


.316 


.284 


.142 


.258 


XI 1 


.062 


.275 


.347 


.277 


.102 


XI 2 


.139 


.351 


.351 


.109 


.189 


XI 3 


.139 


.333 


.333 


.111 


.222 


XI 4 


.072 


.315 


.315 


.153 


.216 


Weighted 


Score 


.295 


.280 


.213 


.212 


Ranking 




1 


2 


3 


4 



3. Advantages & Disadvantages of the Eigenvector Model 
The advantage of the eigenvector model is that it 
facilitates a review of a decision makers consistency 
throughout the analysis by conducting an inspection of 
inverse relationships between respective paired comparisons. 
However, the eigenvector model ignores attribute 
interdependence and does not reflect risk or uncertainty 
involved with the various alternatives in the analysis. In 
the course of a large scale analysis pairwise comparisons 
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can also become quite difficult when the number of pairs is 
increased to a large number. (Shoval and Lugasi,1987) 

C. THE MOLT I -ATTRIBUTE OTILITY MODEL 
1. Description 

Like the additive weight and eigenvector models the 
multi-attribute utility model requires a list of applicable 
attributes to be evaluated. While benefits of information 
systems are not unidimensional, it is probable that 
dimensionality can be contained or at least reduced by 
including only the most relevant attributes to the systems 
being evaluated (Ahituv,1980) . The multi-attribute utility 
model itself has two variations, an additive model and a 
multiplicative model (Formulas 4 and 5 respectively). The 
variants differ in the way they treat risk. (Shoval and 
Lugasi ,1987 ) 

a. Mul ti-Attribute Utility Formulas 

(4) 

1^1 



n 



u. 



i'P (l+JcJc^Uy) ) -1 



(5) 



In both Formulas 4 and 5, Uj depicts the utility 
of alternative j. refers to the utility of attribute i 

in alternative j and U itself is constrained to be between 0 
and 1 by the constant k. Formula 6 below is used to 
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determine k. k^ here portrays the weight of attribute i. 
(Shoval and Lugasi,1987) 



n 

1+Jc*ri 

2=1 



( 6 ) 



b. Model Selection 

The use of either the additive or multiplicative 
model is dependent on the decision maker's outlook on risk. 
The decision aid for that determination is the summation of 
all the attribute weights. If this sum is equal to 1, then 
the decision maker is indifferent to risk and the additive 
model is selected. If the sum is greater than 1, then the 
multiplicative model is selected and k set between 0 and -1. 
When the summation of attribute weights is less than 1, then 
the multiplicative model is also used, yet in this case k is 
set to a value greater than 0. (Shoval and Lugasi,1987) 

c. Application of the Mul ti-Attribute Utility Model 
The multi-attribute utility model requires both 

utility and preference independence of the various 
attributes. Once the attribute independence has been 
confirmed the evaluation of the utility function for every 
attribute can then be based on two axioms, transitivity and 
continuity. The transitivity axiom states that if 
alternative is favored over aj, and aj is favored over aj^, 
then alternative a^ is superior to alternative a^. The 
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continuity axiom refers to the state that if alternative 
is favored over aj, and a^ is favored over a^, then a gamble 
on a^ and a^ can be performed so that there will be a point 
where the decision maker will be indifferent between 
choosing the gamble or being assured of receiving 
alternative aj. If the axioms can be met then the utility 
function can be found through the multi-attribute utility 
model. (Shoval and Lugasi,1987) 

d. Forming the Otility Function 

To form the utility function for each attribute, 
with the above axioms previously being met, the two extreme 
points of the utility curve must be defined. The highest 
level for attribute i is designated as i* and C7(i*) is set 
to equal 1. The lowest level of attribute i is represented 
by i^ and U(i^) is set to 0. i is designated as i* when 
conducting the gambling technique in evaluating the utility 
of a particular level of i. The decision maker is then 
presented with the following situation. Given two 
alternatives, A and B, with all attributes between the two 
alternatives, other than i, being equal. In alternative A 
there is a 100% probability that attribute i will be i'. 
Alternative B is the gamble with a probability of p that 
attribute i will be i* and a probability of 1-p that i will 
be i^. The decision maker is then asked to determine at 
what level of p he would be indifferent between the two 
alternatives. When the decision maker is indifferent to the 
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two alternatives the utility of alternative A equals the 
expected utility of alternative B, or in other words U(i') = 
pUCi*) + (l-p)U(i®). The value of U(i’) is then known to be 
p, for U(i*) is 1 and U(i^) is 0. Other utility points for 
attribute i are found in the same manner by varying the 
level of i'. The utility curve for i is formed by 
connecting the utility points. A continuous curve is not 
required for it is suitable to evaluate only a range of 
relevant values for each attribute. (Shoval and 
Lugasi ,1987 ) 

e. Determining the Attribute freights 

The attribute weights, k^, are found in the same 
way that the attribute utility points were found. To 
define the attribute weights the decision maker is given a 
situation with two alternatives, A and B. This time 
alternative A has attribute i at the highest level, or i*, 
and the remaining attributes at their lowest level, or i^. 
Alternative B is again a gamble with a probability of p that 
all of the attributes are at their respective highest level, 
and a probability of l**p that all of the attributes are at 
their lowest level. The decision maker is then asked to 
choose between the two alternatives by answering the 
following question: "At what level of p would you be 
indifferent between the two alternatives?" The value of 
is then known to be p by the same reasoning that U(i’) is 
equal to p. With the utility function and weights found for 
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every attribute the appropriate version of the model can 
then be applied. (Shoval and Lugasi,1987) 

2. Example of the Multi-Attribute Utility Model 

The example cited here is from Shoval and Lugasi 
(Shoval and Lugasi, 1987) and is the same situation as given 
in the examples for the additive weight and eigenvector 
models . 

a. Determining Attribute Independence 

Prior to the application of the multi-attribute 
utility model the attribute utility and preference 
independence had to be verified. The utility independence 
was confirmed through the repeated use of the gambling 
technique. In each trial the level of the attributes other 
than i were set to different levels. Utility independence 
was known to be true as U(i') remained constant throughout 
the trials. (Shoval and Lugasi, 1987) 

In a similar manner the preference independence 
was also found. Attributes were evaluated taking them two 
at a time. In the first alternative, the first attribute, 
quality of documentation, was set at 2 out of a five-point 
scale and the second attribute, memory capacity, was placed 
at 768K. The second alternative had quality of 
documentation at 4 and memory capacity at i*. All other 
attributes were at their highest respective levels. The 
decision maker was then asked to determine a value for i' 
where he would be indifferent between the two alternatives. 
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This process was repeated with all of the attributes other 
than quality of documentation and memory capacity set to 
their lowest respective levels. As i' was found not to 
change between the two trials, the preference between the 
two attributes was determined to be independent. The rest 
of the attribute pairs were examined in the same way and 
found to be preference independent. (Shoval and 
Lugasi , 1987 ) 

b. Determining Attribute Utility 

In evaluating the utility of the attributes the 
categories were ignored and the 14 attributes were addressed 
in one list. Continuous utility curves were not calculated 
for it was determined that for each attribute only four 
values, one for each alternative, needed to be evaluated. 

To evaluate the memory capacity the decision maker was again 
given two alternatives. Alternative A is certain to receive 
768K capacity. Alternative B is a gamble with the 
probability of p that the capacity will be 1060K and a 
probability of 1-p that the capacity will be 576K. The 
minimum acceptable memory capacity having previously been 
defined at 576K and the highest capacity between all 
alternatives being 1060K. After asking the decision maker 
for a value of p where he would be indifferent between the 
two alternatives he responded that p would have to be 0.3 
for him to be indifferent. Thus the memory capacity of 768K 
has the utility value of 0.3. The utility values of all of 
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the 14 attributes were found using this same method. The 
utilities found for each of the attributes of each of the 
four alternatives are presented in Table 5. (Shoval and 
Lugasi , 1987 ) 

c. Determining Attribute freights 

Attribute weights, k^, were found as in the 
following example for memory capacity. The decision maker 
being provided with two alternative computer systems. One 
has a guaranteed capacity of 1060K and all the remaining 
attributes are at their lowest levels. The other 
alternative is the gamble with a probability of p that all 
attributes are at their highest levels and a probability of 
1-p that all of the attributes are at their lowest levels. 
The decision maker was then asked for a value of p where he 
would feel indifferent between the two alternatives. The 
response was that p would have to be 0.08 for the decision 
maker to be indifferent. Thus the weight of the attribute 
memory capacity is 0.08. The weights for the other 
attributes were found in the same manner and are given in 
Table 5. (Shoval and Lugasi, 1987) 

d. Implementing the Model 

As the sum of the weights was greater than 1 the 
multiplicative utility model. Formula 5, was used in this 
example. The constant k was found to be -0.44268 from 
Formula 6. The ranking of the alternatives is given in the 
last line of Table 4. (Shoval and Lugasi, 1987) 
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Table 5. Utility Values and Weights 



Attribute 


Weight 


A 


Alternatives 
B C 


D 


XI 


0.08 


0.6 


0.6 


0.6 


0.1 


X2 


0.07 


1.0 


1.0 


1.0 


0.5 


X3 


0.04 


0.9 


0.9 


0.8 


1.0 


X4 


0.12 


1.0 


0.9 


0.7 


0.9 


X5 


0.07 


1.0 


0.9 


1.0 


0.8 


X6 


0.07 


1.0 


1.0 


1.0 


0.8 


X7 


0.15 


1.0 


1.0 


1.0 


1.0 


X8 


0.10 


1.0 


1.0 


1.0 


0.9 


X9 


0.15 


1.0 


0.9 


0.7 


0.8 


XI 0 


0.10 


1.0 


0.9 


0.6 


0.9 


XI 1 


0.05 


0.8 


0.7 


1.0 


0.9 


XI 2 


0.09 


1.0 


1.0 


0.9 


1.0 


XI 3 


0.08 


1.0 


1.0 


0.7 


0.8 


XI 4 


0.12 


1.0 


1.0 


1.0 


0.9 


Utility 




.973 


. 944 


.882 


.905 


Ranking 




1 


2 


4 


3 



3. Advantages & Disadvantages of Multi-Attribute 

Utility 

There are three primary advantages of the multi- 
attribute utility model. First, the model is normative in 
that it is based on the axioms of transitivity and 
continuity which reveal the behavioral rules of the decision 
maker. Secondly, the model verifies attribute independence. 
Finally, the model differs from the additive weight and 
eigenvector models in that it has the capacity to take into 
account the issues of risk and uncertainty. (Shoval and 
Lugasi , 1987 ) 

The disadvantages of the multi-attribute utility 
model are two fold. First, the decision maker may have 
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difficulty with the gambling technique in making a 
preference between the two alternatives. Secondly, the 
rules of attribute independence, necessary for the 
implementation of the model, are not always met. Thus there 
are situations were the model can not be used. (Shoval and 
Lugasi , 1987 ) 

D. SUMMARY 

In summary the benefit comparison methods; additive 
weight, eigenvector, and multi-attribute utility, are all 
similar in nature. All three methods start by defining the 
various advantageous features, or non-cost factors, of an 
information system that are to be evaluated and separate 
those features into distinct attributes. Then each 
attribute is assigned either an importance weight or 
relative utility, dependent on the analysis method. A 
score, or utility, is next found for each alternative by 
attribute. From these scores the various formulas of the 
different methods can then be used to arrive at a final 
ranking of each alternative. The methods differ, of course, 
by the formulas used and if any special conditions have to 
be met, such as attribute independence in the multi- 
attribute utility model. 

1. Advantages & Disadvantages of Benefit Comparison 
All of the methods addressed in this chapter have 
their own merits. The additive weight model has the 
capacity for sensitivity analysis however, in the 
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application of the model in the example, the model does not 
take into account risk or uncertainty. The eigenvector 
model has the capacity to verify a decision maker's 
consistency, yet it too does not consider risk or 
uncertainty. In addition, the eigenvector model overlooks 
the issue of attribute interdependence. The multi-attribute 
utility model has the advantages that it is; normative, it 
examines attribute independence, and it takes into account 
risk and uncertainty. The disadvantage of the multi- 
attribute utility model is that the gambling technique, used 
in the differentiation of alternatives, may be difficult for 
some decision makers. The multi-attribute utility model 
also can not be applied in all situations particularly when 
the rules of attribute independence can not be met. 

2. Benefit Comparison and Non-Cost Factor Analysis 
The different functional and risk factors 
recommended by GSA could be selected as the attributes to be 
evaluated in the application of a benefit comparison method 
for the analysis of the non-cost factors. However, by 
including the functional factor of user acceptance in any 
one of the models would require a scoring technique to 
assess user satisfaction with the proposed alternative. 

None of the benefit comparison methods discussed here 
address how such a score could be determined. Thus there 
remains a need for an analysis method or tool that can be 
applied in the evaluation of user acceptance with respect to 
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the analysis of alternatives. Tools that have been 
reconunended for quantifying user acceptance fall under the 
heading of measurements of user satisfaction, which is the 
subject of Chapter V. 
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V. MEASURING USER SATISFACTION 



In the preceding chapter the measurement of user 
satisfaction was introduced as a means of augmentation to 
the benefit comparison methods so that all of the GSA 
stipulated non-cost factors may be evaluated in an analysis 
of alternatives in the acquisition of ADP systems for the 
federal government. The supporters of user satisfaction 
analysis techniques believe so strongly in their work that 
they consider that these techniques should be used more than 
just for augmentation of other evaluation methods 
(Baroudi , Ives , and Olson, 1983). These defenders of user 
satisfaction regard the efforts to measure actual 
productivity benefits of information systems as difficult 
and even futile (Baroudi , Ives, and Olson, 1983). Others 
insist that when dealing with information systems it is 
impossible to derive useful measures such as return on 
investment or pretax profits (Olson, and Neill, 1989). User 
satisfaction, in the view of these authors, is measurable 
and for that fact should be used as the method of choice in 
the analysis of information systems benefit (Baroudi, Ives, 
and Olson, 1983). Methods that can be used in the analysis 
of user satisfaction that will be addressed are: the Bailey 
and Pearson tool, the Electre model, and the multi-attribute 
linear value model. 
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A. USER SATISFACTION ANALYSIS METHODS 



Primarily as a substitute to the objective methods of 
information system benefit analysis, such as net present 
value, a subjective measure of user satisfaction has been 
offered. The technique of user satisfaction, also known as 
user information satisfaction, centers around the use of 
surveys or questionnaires to measure the satisfaction level 
that the users have with their information system. 

Different types of surveys have been suggested, such as 
single item and multiple item. Surveys of single items or 
functions are of limited utility (Baroudi , Ives , and 
Olson, 1983). Multiple item surveys, more broad in scope, 
take a better sample of the users opinions of the 
information system and therefore dominate as preferred 
measurement techniques of user satisfaction (Baroudi, Ives, 
and Olson, 1983). Completed surveys are used to make 
inferences about the beneficial value of an information 
system as a whole. These deductions have been made in the 
past from numerical scales and formulas that have been 
devised by different developers of user satisfaction 
measurement tools. 

B. THE BAILEY AND PEARSON TOOL 

Bailey and Pearson (Bailey and Pearson, 1983) present a 
multiple item survey and a respective scoring method to 
measure computer user satisfaction. They identified 39 
different factors of user satisfaction (see Appendix B). 
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Each factor identified was assigned four distinct adjective 
pairs to assess the factor. The documentation factor, for 
instance, was assigned the distinct adjective pairs of: 

- clear vs. hazy 

- available vs. unavailable 

- complete vs. incomplete 

- current vs. obsolete 

In addition to the four distinct pairs of each factor the 
adjective pair satisfactory - unsatisfactory was included as 
a consistency check on the other adjective pairs. Finally 
the adjective pair important - unimportant was included to 
assign weights of importance to the various factors (see 
Appendix B). (Bailey and Pearson, 1983) 

1. Numerical Scales 

To quantify the results of the survey Bailey and 
Pearson (Bailey and Pearson , 1983 ) derived their own 
importance and satisfaction scales. The importance scale, 
used to designate weights of importance to the respective 
factors, was assigned a value range of 0.10 to 1.00 in 0.15 
increments. The values were associated with importance and 
unimportance as shown below: 

1.0 extremely important 
.85 quite 
.70 slightly 
.55 equally 
.40 slightly 
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.25 quite 

.10 extremely unimportant 
The satisfaction scale below was used to assign 
values to the first four adjective pairs of each factor 
(Bailey, and Pearson, 1983): 

3 extremely satisfied 
2 quite 
1 slightly 
0 equally 
-1 slightly 
-2 quite 

-3 extremely dissatisfied 
2. Bailey and Pearson's Formula 

Through the use of the importance and satisfaction 
scales the perception of a user towards the respective 
information system can be captured as a ntimerical score. 
This value is obtained by first taking a simple arithmetic 
mean of the values I assigned to each of the first 4 
adjective pairs k of each of the factors j using the 
satisfaction scale. Then the summation of the individual 
factorial scores multiplied by their respective weights W, 
from the importance scale, produces the overall measure of 
satisfaction S for each user i as seen in Formula 7. 
(Bailey, and Pearson, 1983) 
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39 « 4 



( 7 ) 



Formula 7 is then normalized to reduce the 
undesirable effects of any factors assessed as neutral or 
meaningless. Any factor is eliminated from consideration if 
all four of the respective distinct adjective pairs were 
evaluated as equal or zero. (Bailey, and Pearson, 1983) 

3. Disadvantages of Bailey and Pearson's tool 

The tool developed by Bailey and Pearson takes a 
broad look at the information system being evaluated and the 
organizational setting in which it was used. However, some 
problems have been identified with the manner in which the 
tool was developed. First, a sample size of only 29 was 
used to validate the tool. Second, each of the 29 subjects 
had participated in the development of the tool (Baroudi, 
Ives and Olson, 1983). Baroudi, Ives and Olson (Baroudi, 

Ives and Olson, 1983) also found that Bailey and Pearson's 
weighted and unweighted results were highly correlated and 
as such the adjective pair important - unimportant was 
unnecessarily included in the questionnaire. This means 
that questions concerning the importance of individual 
factors did not have a significant impact on the final 
results of the measurement tool. A key problem with the 
tool is that it violates measurement theory by treating 
ordinal measures, the adjective pairs, as interval measures 
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such as distance (Galletta and Lederer , 1989) . In the 
application of Formula 7 Bailey and Pearson are conducting 
parametric statistical analysis on ordinal measures which is 
a breach measurement theory (Galletta and Lederer , 1989 ) . 

C. THE HULT I -ATTRIBUTE LINEAR VALUE PROCEDURE 

A procedure that has been employed in an attempt to 
assess the value of information systems is the multi- 
attribute linear value (MALV) technique (Epstein and 
King, 1983). This application of MALV produces more than a 
ranking of alternatives, similar to the Bailey and Pearson 
tool , and though very objective sounding in name this use of 
MALV is in actuality a subjective measure of user 
satisfaction for the data manipulated by the procedure was 
gained through the user survey process (Epstein and 
King, 1983) . 

1. Description 

The application of the MALV procedure by Epstein and 
King uses a survey, numerical scales, and a formula in a 
similar manner as Bailey and Pearson's measurement tool of 
user satisfaction. MALV in this application receives data 
through the use of a ten item survey. Each item is a 
factor, or attribute, of information value. All ten 
attributes were selected by the model designer. The ten 
attributes are: (Epstein and King, 1983) 

Reporting cycle 
Sufficiency 
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Understandabil ity 
Freedom from bias 
Reporting delay 
Reliability 
Decision relevance 
Cost efficiency 
Comparability 
Quantitativeness 

2. Epstein and King's Numerical Scales 

Epstein and King then used a scale to attach 
numerical scores to each of the attributes. The scale 
applied to the reporting cycle measures the frequency that 
information inputs have to be generated for user's decision 
making (Epstein and King, 1983): 



Relative importance weights are obtained by a graphic 
scale on the survey where the response positions are 
stressed and not the scale values. Survey evaluators can 
then quantify the weights by assigning values that match the 
distance ratios between points and the origin. (Epstein and 
King, 1983) 



Reporting Cycle Attribute Scale 



Greatest Satisfaction Possible 

Extremely frequent 

Fairly infrequent 

Moderately frequent 

Very frequent 

Very infrequent 

Lowest Satisfaction Possible 



100 

90 

70 

50 

30 

10 

0 
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3. MALV Formula 



To find the overall value V of an information system 
the multi-attribute linear value model simply multiplies 
each attribute score V(a) by the respective importance 
weight F/ and adds together all of the results. Formula 8 
summarizes this process. (Epstein, and King, 1983) 



10 



1=1 



( 8 ) 



4. Subjectivity in Disguise 

This application of MALV is essentially a subjective 
measure of user satisfaction as the data are acquired 
through the use of a ten item survey. The attribute scales 
and the formula of MALV conflict with measurement theory in 
the same manner as Bailey and Pearson's tool. The model 
fits adjectives to a numerical scale and then analyzes the 
results as if they were interval level measures. The 
validity of the model results are questionable in that they 
are derived from the manipulation of ordinal measures and 
the only legitimate statistical analysis on ordinal measures 
is nonparametric, such as rank ordering (Galletta and 
Lederer,1989) . 
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D. THE ELECTRE MODEL 



1. Description 

A second analysis method of user satisfaction that 
can be used to evaluate completed user satisfaction surveys 
is the Electre model of multiple criteria decision making. 
This model is based on pairwise comparisons of alternatives, 
i and j, and employs two indexes to determine a final 
ranking of various alternatives (Zeleny,1982) . The first 
index is the concordance index C(i,j) which is the summation 
of the corresponding criteria weights for all criteria in 
which i out performs j, divided by the summation of all the 
criteria weights of the model (Bui, 1981). The second index 
is the discordance index D(i,j) which is found by taking the 
largest difference in evaluation scores for which j outranks 
i and dividing that value by the maximinn range between 
highest and lowest possible scores (Bui, 1981). The two 
indexes are then used in conjunction with threshold values 
selected arbitrarily by the user. The concordance threshold 
P ranges from 0.5 to 1.0 and increases in severity as it 
approaches 1. The discordance threshold Q ranges from 0 to 
1 and increases in severity as it approaches 0. The final 
ranking of alternatives is made as follows (Bui, 1981): 



If 

C(i,j) >- P and D(i,j) <= Q 
C(j,i) >= P and D(j,i) <= Q 
i outranks j and j outranks i 
In all other cases 



Then 

i outranks j 
j outranks i 
i and j are equal 
i and j are incomparable 
or the comparison is not 
determinable. 
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2. Example 

a. Applying the El ectre Model 

If the Electre model were applied to the 
analysis of user satisfaction for three alternatives, the 
first issue to be settled would be to select the different 
criteria of user satisfaction that would be evaluated in the 
analysis. Along with the criteria, a weight of relative 
importance for each criterion must also be determined. For 
this example the following criteria and weights will be 
used: 

Understandabi 1 ity of system requests and responses (.2) 
Ease of learning the new system (.3) 

Key mapping, or function keys having the same use as in 
other applications used by the same user (.1) 
Availability of the system to the user (.1) 

System response time (.3) 

A second requirement in setting up the Electre 
model is selecting a scale for use in the application of the 
model. For this example the following scale will be used: 



Greatest Satisfaction Possible 100 
Extremely Satisfied 90 
Fairly Satisfied 70 
Moderately Satisfied 50 
Fairly Dissatisfied 30 
Extremely Dissatisfied 10 
Lowest Satisfaction Possible 0 



With the criteria and scale established for the 
application of the model the scores from a user satisfaction 
survey can be put to use in determining a ranking of the 
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various alternatives. Prior to ranking alternatives the 
concordance and discordance thresholds must be selected. 

Here the values of P= . 6 and C= . 3 are used for the respective 
thresholds. For this example the scores for the different 
alternatives, given by criteria, are displayed in Table 6. 



Table 6. Sample Criteria Score 







Alternative 


Criteria 


1 


2 


3 


Understandabil ity 


90 


70 


30 


Ease of learning 


70 


70 


50 


Key mapping 


70 


50 


10 


Availability 


30 


50 


50 


Response time 


50 


30 


50 



b. Ranking Alternatives 

From the example C(l,2) = (.2 +.1 +.3)/l = .6 
and D(l,2) = 20/100 = .2, therefor C(l,2) = P and D(l,2) < Q 
so that alternative one outranks alternative two. Likewise, 
C(l,3) = P and D(l,3) < Q so that alternative one also 
outranks alternative three. In the comparison of 
alternatives two and three, C(2,3) = P and D(2,3) < Q so 
that alternative two also outranks alternative three. The 
final ranking of alternatives is then: 1, 2, and 3. The 
concordance and discordance index figures for this example 
are given in Table 7. 
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Table 7. Concordance and Discordance Figures 




E. SUMMARY 

User satisfaction analysis is based on user satisfaction 
surveys or questionnaires as illustrated by Bailey and 
Pearson's questionnaire on computer user satisfaction. The 
results of such surveys are interpreted through the use of 
formulas or models such as Bailey and Pearson's formula, the 
Electre model, or MALV. Prior to the application of these 
formulas the responses to the surveys have to be quantified 
though the use of numerical scales such as the importance 
and satisfaction scales of the Bailey and Pearson tool. 
Measurement tools which may be applied objectively, may be 
subjective measures of user satisfaction if the data the 
tool is based on is obtained through surveys of user 
satisfaction . 

1. Precautions of User Satisfaction Analysis Methods 
Analysts employing measure of user satisfaction 
should be cautious for the results may be undesirable for 
the several reasons. First, the adjective pairs and the 
scales used with them have different meanings to the 
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different respondents of the same questionnaire (Galletta 
and Lederer, 1989). Second, the overall score produced by 
some techniques through the summation of responses to 
specific functions of an information system is invalid 
(Galletta and Lederer ,1989) . The functions are assumed to 
be homogeneous when in fact they are heterogeneous which 
makes a score based on their summation illogical (Galletta 
and Lederer , 1989) . If the factors overlap, or are not 
independent of each other, then a tally of the various 
functions simply does not make sense. Overall scores are 
also arrived at without questioning the users’ overall 
perception of the information system which reinforces a 
concern that measures of user satisfaction are unsound 
(Galletta and Lederer , 1989) . Analysts need to avoid methods 
that violate measurement theory by the mannor in which the 
methods call for parametric statistical analysis on ordinal 
measures such as adjective pairs used in surveys. Lastly, 
officials administering the survey can sway the results by 
influencing the setting or condition in which the survey is 
given (Galletta and Lederer , 1989) . 

2. Benefits of User Satisfaction Analysis 

Using the Electre model to rank alternatives is a 
legitimate use of ordinal measures. As such the results of 
the Electre model can augment the findings of other analysis 
methods in the analysis of the non-cost factors, 
particularly user acceptance. Thus there is a method that 
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can be used for the analysis of user acceptance as required 
by GSA for the analysis of alternatives in the acquisition 
of federal ADP systems. 
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VI. DISCUSSION AND CONCLUSION 



In the proceeding chapters different approaches to 
benefit analysis were discussed that could be used in the 
analysis of the different non*cost factors that are required 
by the GSA to be evaluated during an analysis of 
alternatives prior to the federal acquisition of an ADP 
system. The two approaches reviewed were benefit comparison 
and user satisfaction. Both of these strategies have their 
own advantages, disadvantages, and possible applications 
within the DoD and the federal government for the evaluation 
of proposed information systems. 

A. BENEFIT COMPARISON 

1. Advantages of Benefit Comparison 

The benefit comparison methods reviewed were: 
additive weight, eigenvector, and multi-attribute utility 
models, and each have some capacity for sensitivity 
analysis. They also have the advantage that they are 
objective methods of benefit analysis as apposed to user 
satisfaction. As mentioned the multi-attribute utility 
model is normative as it is based upon different axioms. 

2. Disadvantages of Benefit Comparison 

A disadvantage of the benefit comparison methods is 
that additive weight and eigenvector models do not have the 
capacity to consider risk or uncertainty when utilized with 
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the attributes given in Table 1. Yet even more critical is 
that these methods can not be used in all situations. As 
addressed, the mul ti*attribute utility model is incompatible 
with a situation where the attributes are interdependent. 

All of the benefit comparison methods also require that an 
investment of time and effort be made to defining the 
advantages to be gained from the various proposed system so 
that those desired advantages can be used as attributes in 
the analysis. 

3. Feasible Applications of Benefit Comparison 

Benefit comparison techniques are most applicable in 
situations where the users are willing and able to devote 
the time and expense of defining the desired benefits of a 
proposed information system. For a small application that 
can be developed in-house there may not exist the need for a 
detailed benefit analysis. Benefit comparison techniques 
may however, be quite suitable for the evaluation of large 
information systems dealing with administrative activities 
or logistics that are not excluded from the Brooks Act and 
where the GSA recommended non-cost factors must be evaluated 
in the analysis of alternatives for a federal acquisition of 
an ADP system. The GSA functional and risk factors can then 
be used as the attributes in the application of any one of 
the benefit comparison method discussed. 
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B. USER SATISFACTION 



1. Advantages of User Satisfaction 

The main advantage of the user satisfaction is that 
through this technique it is easy to collect data on the 
users' perceptions of an existing information system. This 
same concept can be applied to gaining information on 
prototypes of proposed information systems. The ease of 
data collection is in reference to the fact that it is 
easier to measure a respondent's answers to a questionnaire 
than finding and proving a cause and effect relationship 
between a system capability and any specific organizational 
benefit. The advantage of easy data collection, used in 
conjunction with such methods as the Electre model, enables 
an analyst to employ user satisfaction to rank order several 
competing alternatives, and to do all of this from a users 
point of view. 

2. Disadvantages of User Satisfaction 

A disadvantage of user satisfaction is that the 
technique has been used incorrectly, as in the Bailey and 
Pearson instrument. Analyst should be cautioned when using 
user satisfaction not to confuse ordinal measures, such as 
adjective pairs, as interval measures, such as time or 
distance. Analysts should also question methods that score 
alternatives through the summation of interdependent 
attributes . 
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other disadvantages of the measures of user 
satisfaction include having the responses to the 
questionnaires affected by the manner in which the surveys 
were administered. Inaccurate responses may also be caused 
by what is seen to the respondent's as confusing wording 
used in the questionnaire. Another disadvantage centers 
around the evaluation of proposed system alternatives versus 
evaluating existing systems. Of the literature cited in 
Chapter V on user satisfaction all of the surveys used were 
intended for existing systems. While this technique may be 
selected by some to evaluate system effectiveness, the user 
satisfaction method of benefit analysis appears to be of 
little use in the analysis of proposed information systems 
unless the analysis is tailored so that it can be conducted 
on a prototype of the system. 

3. Feasible Applications of User Satisfaction 

One arena for the application of user satisfaction 
as a technique of evaluating proposed information systems is 
to survey test bed organizations working with prototypes of 
the different alternatives for new proposed system. These 
organizations could receive experience with the different 
prototypes and would be knowledgeable users who could make 
informed responses to surveys on the competing alternatives. 
Using measurements of user satisfaction in a test bed 
organization would be useful for systems that have a high 
degree of user interface, such as in decision support 
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systems or avionic software. The members of these type of 
organizations can offer significant improvements to reduce 
ambiguity through their recommendations concerning: screens 
lay outs, system response messages, or on-line help. Due to 
the large number of units in the military that perform 
similar actions, this technique could be used to evaluate 
highly interactive systems, from the users perspective, 
prior to a large scale purchase. Again, avionic or some 
weapon system software is a good example where the economies 
of scale for the military could make this type of 
prototyping really pay off. This use of prototyping may 
help preclude the purchase of inappropriate or insufficient 
software . 

C. CONNECTING BENEFITS TO ORGANIZATIONAL GOALS 

Buss (Buss, 1983) recommended taking into consideration 
how well an alternative is in line with the organizational 
objectives. This is a vital concern and should be addressed 
by analysts when conducting a benefit analysis. By 
initiating a benefit analysis with a review of 
organizational goals an analyst should be aware of the goals 
that recommendations should address and take into account. 
The organization's goals can also be used as a guide to 
narrow in on what system attributes need to be evaluated 
during the benefit analysis. In Seidmann and Arbel 
(Seidmann and Arbel, 1984) a process was presented that 
hierarchically decomposed the problem of defining system 
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attributes. First, the overall organizational objectives 
were reviewed to determine what type of systems are needed. 
In this example a business was examined and it was 
determined that it was in need of an accounting information 
system. Second, the organizational needs were studied in 
respect to the proposed system. This step looked at major 
functions of the planned system, such as accounts 
receivable, payroll, general ledger, and inventory control. 
Third, specific system operations were addressed that were 
required to support the major functions defined earlier. 

Here such operations were identified as file updating and 
report generation. Finally, the system attributes were 
defined that were required to support the specific system 
functions determined in the previous step. Some of the 
attributes identified in this step were: CPU capacity, main 
memory size, compatibility of software, and vendor support. 
(Seidmann and Ar be 1,1 98 4) 

D. DIFFERENT APPROACHES USED TOGETHER 

There are advantages to using the different approaches 
to benefit analysis in concert with each other and 
capitalizing on the strong points of each. Where system 
capabilities could easily be measured in dollar values, 
present value analysis could be employed to take advantage 
of that method's widely understood methodology. Where 
system benefits are difficult to quantify in dollar term 
benefit comparison methods, such as additive weight, could 
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be used to take advantage of their objective nature and 
capability for sensitivity analysis. The Electre model 
could be used to rank order the alternatives in the analysis 
of user acceptance of a prototype where not only are 
benefits difficult to quantify in dollar term, but where 
analysts need to be cautious with their use of ordinal 
measures in an analysis. By conducting an analysis that 
includes: present value analysis of cost factors, using the 
additive weight model to analyze non-cost factors, and using 
the Electre model to specifically analyze user acceptance, a 
more complete picture can be presented in the analysis of 
alternatives. This will allow for a more informed decision 
in the final selection of alternatives. Both the Eletre 
model and present value analysis could be applied toward 
this type combined analysis without modification. The 
additive weight model would only require slight changes from 
the example given previously. 

E. ADDITIVE WEIGHT MODEL IN FEDERAL ACQUISITIONS 

The functional and risk factor defined by the GSA would 
be used in the application of the additive weight model for 
the analysis of non-cost factors. The functional factor of 
user-acceptance would be excluded from analysis under the 
additive weight model and reserved for the Electre model so 
as to preclude the wrongful use of ordinal measures. The 
break down of non-cost factors in an additive weight 
analysis would resemble Table 8. Starting with these 
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defined attributes and categories an analyst would then have 
to ask the decision maker to assign weights between 0 and 1 
to each of the attributes and categories. The sum of the 
weights assigned to the two categories would have to equal 1 
and the sum of the weights assigned to the attribute of each 
category would also have to equal 1. Sample weights are 
given in Table 8. 



Table 8 . Sample Heights for Non-Cost Categories and 
Attributes 



Category 


Attribute 


Heights 


Z1 


Functional Factors 


.60 








XI 


Obsolescence 




.08 






X2 


Availability 




.08 






X3 


Reliability 




.11 






X4 


Maintainability 




.16 






X5 


Expandabi 1 i ty 




.11 






X6 


Flexibility 




.10 






X7 


Security 




.16 






X8 


Privacy 




.07 






X9 


Personnel Impacts 


.06 






XI 0 


Accountability 




.07 


Z2 


Risk Factors 






.40 








XI 1 


Financial 




.20 






XI 2 


Technical 




.50 






XI 3 


Schedule 




.30 



1. Saunple Scores for the Additive Weight Hodel 

The next step in applying the additive weight model 
is to score the alternatives one attribute at a time. A 
range of 0 to 1 is used for the individual attribute scores. 
If the first 13 attribute scores given in Table 2 are used 
as an example, in combination with the category and 
attribute weights stated in Table 8, then the total weighted 
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score for alternative A would be 0.9272 as expressed in 
Table 9. 



Table 9. Calculation of the Weighted Score for 
Alternative A 



Attribute 


Category x 
Weight 


Attribute 

Weight 


X Attribute 
Score 


= Weighted 
Score 


XI 


0.6 


.08 


0.8 


.0384 


X2 


0.6 


.08 


0.8 


.0384 


X3 


0.6 


.11 


0.8 


.0528 


X4 


0.6 


.16 


1.0 


.0960 


X5 


0.6 


.11 


1.0 


.0660 


X6 


0.6 


.10 


0.8 


.0480 


X7 


0.6 


.16 


1.0 


.0960 


xe 


0.6 


.07 


0.9 


.0378 


X9 


0.6 


.06 


1.0 


.0360 


XI 0 


0.6 


.07 


0.9 


.0378 

.5472 


XI 1 


0.4 


0.2 


0.9 


.0720 


XI 2 


0.4 


0.5 


1.0 


.2000 


X13 0.4 

Total Weighted Score 


0.3 


0.9 


.1080 

.3800 

.9272 



2. Advantage of this Application 

The additive weight's recognized weakness, that of 
not taking into account risk or uncertainty, is counteracted 
in this application of the model. The weakness is cancel 
because the three different risk factors; financial, 
technical, and schedule, are specifically addressed as 
attributes when using the GSA recommended non-cost factors. 
This application of the additive weight model has the 
advantages of being: conceptually simplistic, calculable on 
any standard spreadsheet application, and in accordance with 
the guidelines set forth by the GSA for federal ADP 
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acquisitions. The results of this application of the 
additive weight model combined together with the results of 
present value analysis and user acceptance analysis, through 
the use of the Electre model, offer a more complete 
evaluation of alternatives than any of the methods could 
offer when used alone. 

F. CONCLUSION 

One method of benefit analysis is not suitable for all 
situations where proposed information systems are in need of 
evaluation. Nor is one method alone appropriate for the 
evaluation of all of the factors that must be taking into 
account during an analysis of alternative systems. Each 
analysis method has its own advantages and disadvantages. 
Gains can be made by combining methods to take advantage of 
the respective potentials of each. Using different methods 
together would also provide a more complete analysis as some 
methods used alone are inappropriate for certain 
applications, such attempting to use present value analysis 
in an evaluation that includes the measurement of user 
acceptance. System analysts conducting analysis of 
alternatives for a federal acquisition of information 
systems will have to be knowledgeable of the different 
benefit analysis methods. To obtain the required approval 
for development of information systems, and to abide by the 
GSA requirements for analysis of alternatives, system 
analysis will also have to know how to implement the right 
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mix of benefit analysis methods so to fully evaluate both 
cost and non-cost factors. 
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APPENDIX A: EIGENVECTOR MATRICES 



Figure 2. Pairwise Comparison of Attributes by Categories 
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Figure 3. Pairwise Comparison of Categories 
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Figure 4. Comparison of Alternatives by Attributes 
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APPENDIX B: COMPUTER USER SATISFACTION QUESTIONNAIRE 



Use the following scale in response to each pair of 
adjectives : 



! extremelv!auite!sliqhtlv!eauallv!sliahtlv iauite! extremely! 

1. Top management involvement: The positive or negative 

degree of interest, enthusiasm, support, or participation of 
any management level above the user's own level toward 
computer-based information systems or services or toward the 
computer staff which supports them. 



strong 

consistent 

good 

significant 

satisfactory 

important 



weak 

inconsistent 

bad 

insignificant 

unsatisfactory 

unimportant 



2. Organizational competition with the EDP unit: The 

contention between the respondent's organizational unit and 
the EDP unit when vying for organizational resources or for 
responsibility for success or failure of computer-based 
information systems or services of interest to both parties. 



productive 

rational 

low 

harmonious 

satisfactory 

important 



destructive 

emotional 

high 

dissonant 

unsatisfactory 

unimportant 



3. Priorities determination: Policies and procedures which 

establish precedence for the allocation of EDP resources and 
services between different organizational units and their 
requests . 



fair 

consistent 

just 

precise 

satisfactory 

important 



unfair 

inconsistent 

unjust 

vague 

unsatisfactory 

unimportant 
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4. Charge-back method of payment for services: The 

schedule of charges and the procedures for assessing users 
on a pro rata basis for the EDP resources and services that 
they utilize. 



just 
reasonabl e 
consistent 
known 
satisfactory 
important 



unjust 

unreasonable 

inconsistent 

unknown 

unsatisfactory 

unimportant 



5. Relationship with the EDP staff: The manner and methods 

of interaction, conduct, and association between the user 
and the EDP staff. 



harmonious 

good 

cooperative 

candid 

satisfactory 

important 



dissonant 

bad 

uncooperative 

deceitful 

unsatisfactory 

unimportant 



6. Communication with the EDP staff: The manner and 

methods of information exchange between the user and the EDP 
staff . 



harmonious 

productive 

precise 

meaningful 

satisfactory 

important 



dissonant 

destructive 

vague 

meaningless 

unsatisfactory 

unimportant 



7. Technical competence of the EDP staff: The computer 

technology skills and expertise exhibited by the EDP staff. 



current 

sufficient 

superior 

high 

satisfactory 

important 



obsol ete 
insufficient 
inferior 
1 ow 

unsatisfactory 

unimportant 
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8. Attitude of the EDP staff: The willingness and 

commitment of the EDP staff to subjugate external , 
professional goals in favor of organizationally directed 
goals and tasks. 



user-oriented 

cooperative 

courteous 

positive 

satisfactory 

important 



self-centered 

belligerent 

discourteous 

negative 

unsatisfactory 

unimportant 



9. Schedule of products and services: The EDP center 

timetable for production of information system outputs and 
for provision of computer-based services. 



good 
regular 
reasonabl e 
acceptabl e 
satisfactory 
important 



bad 

irregular 
unreasonabl e 
unacceptabl e 
unsatisfactory 
unimportant 



10. Time required for new development: The elapsed time 

between the user's request for new applications and the 
design, development, and/or implementation of the 
application systems by the EDP staff. 



short 
dependable 
reasonabl e 
acceptabl e 
satisfactory 
important 



1 ong 

undependabl e 
unreasonabl e 
unacceptabl e 
unsatisfactory 
unimportant 



11. Processing of change requests: The manner, method, and 

required time with which the EDP staff responds to user 
requests for changes in existing computer-based information 
systems or services. 



fast 

timely 

simple 

flexible 

satisfactory 

important 



slow 

untimely 

complex 

rigid 

unsatisfactory 

unimportant 
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12. Vendor support: The type and quality of the service 

rendered by a vendor, either directly or indirectly, to the 
user to maintain the hardware or software required by that 
organizational status. 



skilled 

sufficient 

eager 

consistent 

satisfactory 

important 



bungling 

insufficient 

indifferent 

inconsistent 

unsatisfactory 

unimportant 



13. Response/ turnaround time: The elapsed time between a 

user-initiated request for service or action and a reply to 
that request. Response time generally refers to the elapsed 
time for terminal type request or entry. Turnaround time 
generally refers to the elapsed time for execution of a 
program submitted or requested by a user and the return of 
the output to that user. 



fast 

good 

consistent 

reasonable 

satisfactory 

important 



slow 

bad 

inconsistent 

unreasonable 

unsatisfactory 

unimportant 



14. Means of input/output with EDP center: The method and 

medium by which a user inputs data to and receives output 
from the EDP center. 



convenient 

clear 

efficient 

organized 

satisfactory 

important 



inconvenient 

hazy 

inefficient 

disorganized 

unsatisfactory 

unimportant 



15. Convenience of access: The ease of difficulty with 

which the user may act to utilize the capability of the 
computer system. 



convenient 

good 

easy 

efficient 

satisfactory 

important 



inconvenient 

bad 

difficult 

inefficient 

unsatisfactory 

unimportant 
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16. Accuracy: The correctness of the output information. 



accurate 

high 

consistent 

sufficient 

satisfactory 

important 



inaccurate 

low 

inconsistent 

insufficient 

unsatisfactory 

unimportant 



17. Timeliness: The availability of the output information 

at a time suitable for its use. 



timely 

reasonable 

consistent 

punctual 

satisfactory 

important 



untimely 

unreasonable 

inconsistent 

tardy 

unsatisfactory 

unimportant 



18. Precision: The variability of the output information 

from that which it purports to measure. 



sufficient 

consistent 

high 

definite 

satisfactory 

important 



insufficient 

inconsistent 

low 

uncertain 

unsatisfactory 

unimportant 



19. Reliability: The consistency and dependability of the 

output information. 



consistent 

high 

superior 

sufficient 

satisfactory 

important 



inconsistent 

low 

inferior 

insufficient 

unsatisfactory 

unimportant 



20. Currency: The age of the output information. 



good 

timely 

adequate 

reasonable 

satisfactory 

important 



bad 

untimely 

inadequate 

unreasonable 

unsatisfactory 

unimportant 
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21. Completeness: The comprehensiveness of the output 

information content. 



complete 

consistent 

sufficient 

adequate 

satisfactory 

important 



incomplete 

inconsistent 

insufficient 

inadequate 

unsatisfactory 

unimportant 



22. Format of output: The material design of the layout 

and display of the output contents. 



good 
simpl e 
readable 
useful 
satisfactory 
important 



bad 

complex 

unreadable 

useless 

unsatisfactory 

unimportant 



23. Language: The set of vocabulary, syntax, and 

grammatical rules used to interact with the computer 
systems . 



simple 

powerful 

easy 

easy-to-use 

satisfactory 

important 



complex 

weak 

difficult 

hard-to-use 

unsatisfactory 

unimportant 



24. Volume of output: The amount of information conveyed 

to a user from computer-based systems. This is expressed 
not only by the number of reports or outputs but also by the 
voluminousness of the output contents. 



concise 

sufficient 

necessary 

reasonable 

satisfactory 

important 



redundant 

insufficient 

unnecessary 

unreasonable 

unsatisfactory 

unimportant 
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25. Relevancy: The degree of congruence between what the 

user wants or requires and what is provided by the 
information products services. 



useful 

relevant 

clear 

good 

satisfactory 

important 



useless 

irrelevant 

hazy 

bad 

unsatisfactory 

unimportant 



26. Error recovery: The methods and policies governing 

correction and return of system outputs that are incorrect. 



fast 

superior 

complete 

simple 

satisfactory 

important 



slow 

inferior 
incomplete 
compl ex 

unsatisfactory 

unimportant 



27. Security of data: The safeguarding of data from 

misappropriation or unauthorized alteration or loss. 



secure 

good 

definite 

complete 

satisfactory 

important 



insecure 

bad 

uncertain 
incompl ete 
unsatisfactory 
unimportant 



28. Documentation: The recorded description of an 

information system. This includes formal instructions for 
the utilization of the system. 



clear 

available 

complete 

current 

satisfactory 

important 



hazy 

unavailable 

incomplete 

obsolete 

unsatisfactory 

unimportant 
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29. Expectations: The set of attributes or features of the 

computer-based information products or services that a user 
considers reasonable and due from the computer-based 
information support rendered within the organization. 



pleased 

high 

definite 

optimistic 

satisfactory 

important 



displeased 

low 

uncertain 

pessimistic 

unsatisfactory 

unimportant 



30. Understanding of systems: The degree of comprehension 

that a user possesses about the computer-based information 
system or services that are provided. 



high 

sufficient 

complete 

easy 

satisfactory 

important 



low 

insufficient 

incomplete 

hard 

unsatisfactory 

unimportant 



31. Perceived utility: The user's judgement about the 

relative balance between the cost and the considered 
usefulness of the computer-based information products or 
services that are provided. The costs include any costs 
related to providing the resource, including money, time, 
manpower, and opportunity. The usefulness includes any 
benefits that the user believes to be derived from the 
support . 



high 

positive 

sufficient 

useful 

satisfactory 

important 



low 

negative 

insufficient 

useless 

unsatisfactory 

unimportant 



32. Confidence in the system: The user's feelings of 

assurance or certainty about the systems provided. 



high 

strong 

definite 

good 

satisfactory 

important 



low 

weak 

uncertain 

bad 

unsatisfactory 

unimportant 
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33. Feeling of participation: The degree of involvement 

and commitment which the user shares with the EDP staff and 
others toward the functioning of the computer-based 
information systems and services. 



positive 

encouraged 

sufficient 

involved 

satisfactory 

important 



negative 

repelled 

insufficient 

uninvolved 

unsatisfactory 

unimportant 



34. Feeling of control: The user's awareness of the 

personal power or lack of power to regulate, direct or 
dominate the development, alteration, and/or execution of 
the computer-based information systems or services which 
serve the user's perceived function. 



high 

sufficient 

precise 

strong 

satisfactory 

important 



low 

insufficient 

vague 

weak 

unsatisfactory 

unimportant 



35. Degree of training: The amount of specialized 

instruction and practice that is afforded to the user to 
increase the user's proficiency in utilizing the computer 
capability that is unavailable. 



complete 

sufficient 

high 

superior 

satisfactory 

important 



incomplete 

insufficient 

low 

inferior 

unsatisfactory 

unimportant 



36. Job effects: The changes in job freedom and job 

performance that are ascertained by the user as resulting 
from modifications induced by the computer-based information 
system and services. 



liberating 

significant 

good 

valuable 

satisfactory 

important 



inhibiting 

insignificant 

bad 

worthless 

unsatisfactory 

unimportant 
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37. Organizational position of the EDP function: The 

hierarchical relationship of the EDP function to the overall 
organizational structure. 



appropriate 

strong 

clear 

progressive 

satisfactory 

important 



j_ inappropriate 
j_ weak 
j_ hazy 

J_ regressive 
J_ unsatisfactory 
J_ unimportant 



38. Flexibility of systems: The capacity of the 

information system to change or to adjust in response to new 
conditions, demands, or circvunstances . 



flexible 

versatile 

sufficient 

high 

satisfactory 

important 



rigid 

limited 

insufficient 

low 

unsatisfactory 

unimportant 



39. Integration of systems: The ability of systems to 

communicate/transmit data between systems servicing 
different functional areas. 



complete 

sufficient 

successful 

good 

satisfactory 

important 



incomplete 

insufficient 

unsuccessful 

bad 

unsatisfactory 

unimportant 



The above user satisfaction questionnaire is basis of a 
copyrighted measurement instrument and is reprinted here 
only as an educational example. Application of the 
measurement instrument, other than in research, should be 
preceded by the receipt of permission from Dr. Pearson. 
(Bailey and Pearson, 1983) 
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